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Abstract 

A comprehensive formalism for the description of neutrino oscillations in the 
Earth in a general scheme with three massive neutrinos and the mass hierarchy 
mi <^ m2 <^ ma is presented. Using this formalism, which is valid both in 
vacuum and in a medium, the matter effect on the oscillations of low-energy 
neutrinos is discussed, pointing out the existence of very long oscillations which 
are independent of the neutrino masses and the neutrino energy, and are very 
sensitive to the matter density along the neutrino trajectory. As an example 
of application of the formulation, a fit of the Kamiokande atmospheric neutrino 
data with the matter effect taken into account for neutrinos propagating in the 
Earth is presented. The results of the fit indicate that 4 x 10~^eV^ < mg < 
4 X 10~^eV^ and the oscillation amplitudes in all channels {u^ ^ Ue-, i^^i ^ i^r, 
z/g ^ Ur) could be large. Hence, long-baseline experiments with reactor (CHOOZ 
and Palo Verde) and accelerator (K2K, MINOS and ICARUS) neutrinos could 
observe neutrino oscillations in all channels with a relatively large statistics. 
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1 Introduction 



The atmospheric neutrino anomaly |jT|, 
existence of neutrino oscillations I 



1^, ^, 1^ is one of the indications in favor of the 
see Refs.J^, ^ |TU|, 0), a phenomenon which is 



realized if neutrinos are massive particles and leptons, as in the case of quarks, are mixed 
(i.e. the mass matrices of the weak- interacting leptons are not diagonal). Neutrino 
oscillation is considered today one of the most interesting phenomena in high energy 
physics as well as one of the most promising ways to search new physics beyond the 
Standard Model (see Ref.[|, |lTl). 

The atmospheric neutrino anomaly has been observed by the Kamiokande [0, ^, 
1MB [0 and Soudan ||^ experiments, which measured a ratio of contained muon-like to 
electron-like events smaller than the expected one |T3|, |15|, |16|. The experimental 
data can be explained by 



difference Am^ ~ 10 ^eV 



-i> z/t- or Ufj, ^ Ue oscillations with a neutrino mass-squared 
A preliminary analysis of recent SuperKamiokande data 
T7| confirms the atmospheric neutrino anomaly, but indicates a smaller value of Am^: 
3 X 10^^ < Am? ^ 6 X lO^'^eV^. The anomaly observed by Kamiokande and Su- 
perKamiokande cannot be explained with a background flux of slow neutrons 0, |19|, |17 



In this paper, for the sake of definiteness, we use only the Kamiokande data, waiting for 
definitive data from SuperKamiokande. 

On the other hand, no anomaly was observed in the ratio of contained muon-like to 
electron-like events by the Frejus and NUSEX experiments and in the ratio of 
contained muon-like to total events with energy higher than 1 GeV by the 1MB experiment 
2^. However, the statistical error of the Frejus and NUSEX data is higher than that of 



the Kamiokande data and the inclusion in the fit of the data of the Frejus and NUSEX 
experiments does not eliminate the necessity of neutrino oscillations [^, . Also the high 
energy 1MB data have relatively large errors and they are not statistically incompatible 
with the data of the Kamiokande experiment |^ . No anomaly was also observed in the 
absolute flux of upward-going muons by the Kamiokande ||2^ , 1MB p6| , Baksan and 
MACROQ 1^ experiments. However, the knowledge of the absolute value of the flux 
of neutrino-induced upward-going muons without neutrino oscillations is rather model- 
dependent (see Ref.|^ for a combined analysis of the available upward-going muon 
data). 

The neutrino oscillation solution of the atmospheric neutrino anomaly will be checked 
in the next years by the CHOOZ ||31[] and Palo Verde long-baseline Ue disappearance 
experiments with reactor anti- neutrinos and by the KEK-Super Kamiokande (K2K) |]33| , 
Fermilab-Soudan (MINOS) and CERN-Gran Sasso (ICARUS) [g^ long-baseline 
experiments with accelerator neutrinos and anti-neutrinos, which are sensitive to the 



disappearance of'z/^ and to'z/^^Vg and 'z/i— >^z/i transitions. 



Another important indication in favor of neutrino oscillations comes from the results of 



^ Although the total flux of upward-going muons observed by the MACRO experiment is compatible, 
within errors, with the calculated one (without neutrino oscillations), the angular shape of the flux 
does not match the expectations, because of a deficit in the angular bins corresponding to the vertical 
direction. However, the compatibility of this deficit with neutrino oscillations is still uncertain pq|. 
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the solar neutrino experiments (Homestake Kamiokande GALLEX |]39|, SAGE 



40| and SuperKamiokande 0]), which observed event rates significantly smaller than 



the values predicted by the Standard Solar Model (SSM) |^ ^ ^]. Moreover, the 
experimental data indicate that the suppression of solar z/g's depends on energy, excluding 
an astrophysical solution of the solar neutrino problem ||^ . Assuming the validity of the 
SSM, the experimental data can be described by MSW resonant transitions with 
Am^ ~ 10-5 eV^ H or by vacuum oscillations with Am^ ~ 10 ^°eV^ (Am^ is a 
neutrino mass-squared difference). 

A solution of the solar and atmospheric neutrino problem with neutrino oscillations 
requires the existence of two different scales of neutrino mass-squared differences, which 
corresponds to the existence of three massive neutrinos, z/i, z/2 and z/3, with masses mi, 
7712 and m3, respectively. These three massive neutrinos are mixings of the three flavor 
neutrinos, u^, and z/^, whose existence is known[]from the measurements of the invisible 
width of the Z boson done by LEP experiments (see Ref . ||50|| ) . 

The experimental upper bounds on the values of the neutrino masses (see Refs. ^ ) 
imply that the neutrino masses are much smaller than the masses of the corresponding 
charged leptons. An attractive theoretical explanation of this smallness is given by the 
see-saw mechanism |^|, which allows the hierarchical pattern (see Refs.[§, ^, |10|, |TT1 ) 



mi <^ m2 <^ ma . (1.1) 

This is the simplest and most natural scheme for the neutrino masses, analogous to the 
mass-schemes of charged leptons, up and down quarks, all of which have a hierarchy of 
masses. The hierarchical scheme (|1 . 1| ) is exactly what is needed for a neutrino oscillation 
solution of the solar and atmospheric neutrino problems, if ~ 10^^ or 10^^° eV^ (for the 
MSW or vacuum oscillation solution of the solar neutrino problem) and mg ~ 10"^ eV^ 
(for the solution of the atmospheric neutrino anomaly). In this paper we consider this 



possibility |23, E3] and we present a comprehensive formalism for the description of at- 



mospheric neutrino oscillations, by taking into account the matter effect |£2|, ^ for 
neutrinos propagating in the Earth We believe that this formalism will be useful 

for the analysis of the atmospheric neutrino data of the next generation of experiments, 
especially those of the SuperKamiokande experiment. 



Recently the LSND collaboration reported the observation of anomalous z/gp 



n events produced by neutrinos originating from (j/ decays at rest. These events could 
be explained by — > oscillations with Am^ ~ leV^, but this possibility is excluded 
in the schemes with three neutrinos if the neutrino masses have the values appropriate 
for the explanation of the solar and atmospheric neutrino problem [^]. In this paper 
we do not consider the LSND indication in favor of neutrino oscillations, waiting for its 
confirmation by other experiments (KARMEN ||5^ and others |58[] ). 

In Section ^ we present our fit of the Kamiokande data in the scheme ( p. . 1|) , with the 
assumption that ~ 10"^ or 10"^° eV^, so that Amgi = m^—ml ^ m| is relevant for the 

^ The tau neutrino has not been directly observed, but it is widely believed to exist without doubt 
because it is necessary for the consistency of the Standard Model of electro-weak interactions. 
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oscillations of solar neutrinos. With these assumptions, the oscillations of atmospheric 
neutrinos depend only on Am^i = m\ — m\ ~ m|. 

In the analysis of the experimental indications in favor of neutrino oscillations one 
must take into account also the numerous negative results of the terrestrial neutrino 
oscillation experiments with reactor z/g's and accelerator i/^'s and i>^'s (the latest results 
are summarized in Ref.||59||). In particular, in Section |3.3| it will be shown that part of 
the region of neutrino mixing parameters allowed by the atmospheric neutrino data is 
excluded by the results of the Bugey and Krasnoyarsk reactor Vf. disappearance 
experiment. 

The plan of the paper is as follows. In Section Q we present the general formalism 
for the description of oscillations of three neutrinos in vacuum and in matter. In Section 
2ri| we mainly discuss the oscillations of atmospheric neutrinos in the scheme ( |1 . 1| ) with 



three neutrinos and a mass hierarchy. In Section |2.2| we discuss the matter effect on 
the oscillations of neutrinos with low energy. In Section ^ we present our fit of the 
Kamiokande atmospheric data. In Section ^ we draw our conclusions. In the Appendix 
we present a derivation of the parameterization of the mixing matrix used in the paper. 



2 Oscillations of 3 neutrinos 

The theory of neutrino mixing (see Refs.|]^, ||, ||, 0) is based on the hypothesis that 
the left-handed flavor neutrino fields VaL are superpositions of the left-handed components 
of (Dirac or Majorana) massive neutrino fields v^l- 

^Uakl^kL, (2.1) 



where U is a unitary mixing matrix (?7f/^ = f/^f/ = 1). 

The results of the LEP experiments on the measurement of the invisible width of the 
Z boson (see Ref.0]) imply that only three "light" active flavor neutrinos exist in nature, 
i.e. z/e, i^n and Ur- However, the existence of sterile flavor neutrinos is not ruled out and 
the number of neutrinos with a definite mass is unknown. We will consider here the 
simplest scenario with only three active flavor neutrinos and three (Dirac or Majorana) 
neutrinos with definite mass. Hence, in the following we will consider the flavor indices 
a, j3, p,a = e, /i, r and the mass indices k,j = 1, 2, 3. 

A neutrino with momentum p produced by a charged-current weak interaction pro- 
cess together with a charged lepton a is described by a flavor state |z/q,(j9)), which is a 
superposition of neutrino mass eigenstates |z/fc(p)): 

Mp)) = J2u:,Wkip)) . (2.2) 

k 

The state \i^k{p)) describes a neutrino with a definite mass m^, momentum p, energy 
Ek = a/p^ + ml, and satisfies the energy eigenvalue equation 

noMp)) = Ek\uk{p)) , (2.3) 
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where Tio is the free neutrino Hamiltonian. Since the neutrino masses are very small, the 
detectable neutrinos are extremely relativistic and the approximation i^^ ~ p + m\/2p is 
allowed. 

In this paper we consider atmospheric neutrinos, which are produced at the top of 
the atmosphere by the interaction of primary cosmic rays with the nuclei in the air 
(see Ref . [p^ ) . Atmospheric neutrinos propagate in the atmosphere, which is practically 
equivalent to vacuum, and in the interior of the Earth, where the matter density is 
sufficiently high to modify the effective energy-momentum dispersion relation through 
the coherent interaction with the particles in the medium ||5^, In the following 

part of this Section we derive the evolution equation for the transition amplitudes between 
flavor states which is valid both in vacuum and in matter. 

For neutrinos propagating in matter, the Hamiltonian Ti is given by 

n = no + ni, (2.4) 

where Tij is the effective weak interaction Hamiltonian due to the coherent interaction 
with the electrons, protons and neutrons in a medium. The flavor states ( p.2|) are eigen- 
states of Hj: 

niMp)) = Vc,Mp)) , (2.5) 

with 

Ve = Vcc + VMc, V, = Vr = VMc- (2.6) 

Here Vcc and V^c are, respectively, the charged-current and neutral-current effective 
potentials of neutrinos, given by 

/2 

Vcc = V2GfN,, Vnc = -\GfN^, (2.7) 

where Gi? is the Fermi constant and Ne and Nn are, respectively, the electron and neutron 
number densities of the medium^. The effective potentials of anti-neutrinos have the same 
absolute value, but opposite sign: Vcc = — Vc-c*, Vnc = —Vnc- 

Let us consider a neutrino with momentum p produced at the time t = by a charged- 
current weak interaction process together with a charged lepton a {a = e,/i, r). In the 
Schrodinger picture, at the time t this neutrino is described by the state 

\v^-\p,t)) = Y,^'^;\p,t)\u,{p)) . (2.8) 

This state is a superposition of flavor states |z^/3(p)) with amplitudes (p^p\p,t) which 

depend on time and have the initial value (p^^\p, 0) = 6^(3- 

Here we consider an electrically neutral medium, in which the number density of protons is equal 
to the number density of electrons. This implies that the neutral-current effective potentials of protons 
and electrons cancel each other. 
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The time evolution of the state ( p.8| ) is given by the Schrodinger equation 

d 
di 



= Y.^f\p^) \^U;^E,\v,{p)) + V,\v,{p)yi (2.9) 

= 5^^(")(p,t) \ y^u;,EuU,u + v,5,A Mp)) ■ 

(T,p \ k J 

Projecting this equation on (z^/3(p)| and taking into account that {up{p)\up{j))) = 5pp, we 
obtain 



p \ k / 



¥.(°)(p,t). (2.10) 



This is the evolution equation for the flavor amplitudes ip^^\p, t), whose squared-modulus 

(a) ^ 

gives the probability of — > J^is transitions: P^^^^^{p,t) = ipj^'{p,t) . 
For relativistic neutrinos we have 

' i ^^"^ = + ^^^^ ^ ^ if ^ ^""^ ^^"^ ' 

(2.11) 

where we have separated the contribution of the neutral-current effective potential Vatc 
which is the same for the three neutrino flavors, from the contribution of the charged- 
current effective potential Vcc, which affects only the electron neutrino component. The 
time t is equal (in natural units) to the distance L of propagation of the neutrino. 

Since neutrino oscillations are due to the difference of the phases of different mass 
eigenstates, in the following we consider the simplified evolution equation for the ampli- 
tudes 

ij'-;\p,t) ^ ip^;\p,t) exp(^tpt + tj\^c{t')dt'^ , (2.12) 

where we have taken into account the fact that V^c ^lay not be constant along the 
neutrino trajectory. For relativistic neutrinos we have 

' ^ ^^ = ^i^ if ^p*'^ + ^^^) ^J"^^^' ^) • ^2.13) 

This is the evolution equation for the transition amplitudes 'ip^^\p, t), which allows one to 
calculate the oscillation probabilities in vacuum (Vcc = 0) as well as in matter (Vcc 7^ 0). 
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The probability to detect a neutrino with flavor (3 after a time t, which corresponds to 
propagation distance L = t, is given by 



a 



(2.14) 



The solution of the evolution equation (|2.13| ) in vacuum is simply given by 



i^t\p.t) = Y.^:kUpu exp f-.^ 



(2.15) 

The application of Eq. ( p.l4| ) leads to the well known formula for the oscillation probabil- 
ities 

-2— , (2.16) 



k 

with Am^^- = — m^. 

We now derive a method for the solution of the evolution equation (|2.13|) in matter 
It is convenient to write the evolution equation ( |2.13| ) in matrix form: 

'Jt'^w\v.t) = Y^{UM'U^^Ay,)^%^ (P, t) , 



(2.17) 



where 



M = diag(mi, m2, ma) , Aw = diag(Acc, 0, 0) , (2.1^ 



. For anti- neutrinos, Aqc must be replaced by Aqc = —Acc- 
ng matrix U can be written in the form (see the Appendix ^ 



and Acc = '^pVcc- ^or anti-neutrn 
The 3x3 mixing matrix U can 

U = V2sVnW^2D{X) 



with the orthogonal matrices 



(2.19) 



1^23 = 











V 



COS 1923 sin ■^23 

- sin 1?23 cos 1^23 J 



^ cos-i^is sin-i^is 

1 
^— sin?9i3 cos?9i3 j 



(2.20) 



and the unitary matrices 

Wu = Di2 Vu , D{X) = diag(e*^^ , e'^' , l) , 



(2.21) 
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with 



12 



COS 1912 sin'(9i2 0^ 
— sin'(9i2 cos'(9i2 
V 



Di2 = diag(e^''-, 1, 1) 



(2.22) 



Here "(923, ^^is, and "(912 are the three mixing angles, 7712 is the Dirac CP- violating phase 
|6^ and Ai and A2 are the two Majorana CP-violating phases. The matrix Vab represents 
a rotation of an angle dab in the Va^^b plane. In the case of Dirac neutrinos the matrix 
-D(A) containing the two Majorana CP-violating phases can be eliminated with a suitable 
redefinition of the arbitrary phases of the Dirac neutrino fields. This operation is not 
possible for Majorana neutrinos because the Majorana mass term is not invariant under 
rephasing of the neutrino fields. However, the presence of the matrix -D(A) does not have 
any effect on neutrino oscillations in vacuum [|64| , |66[| as well as in matter Indeed, 
from Eq.( |2.17D it is clear that neutrino oscillations depend on the quantity f/M^ U'^ and, 
since the diagonal matrix commutes with the diagonal matrix -D(A), we have 



UM^U^ = V23 Vi3 W12 D{X)M^D{Xy V^.^ 



23 



^23 ^13 W^i2 W V;tj . (2.23) 



Contrary to the Majorana phases, the Dirac phase 7712, contained in Wu, has an effect 

CP 

on neutrino oscillations Under CP transformations U — ^ U*, which is equivalent 

CP 

to rii2 — > —T]i2- This means that CP violation could be observed in neutrino oscilla- 



tions, measuring the differences P^^. 
P„^^„„ = P-y^^D. and therefore P, 



-Pn 



- Pr, 



for a (3 (CPT invariance implies that 
,ufs - - Ufs^uc ...^.^.■^.^ . vc^Pc - uc^^uc, is always equal to zero). Assuming 
CPT invariance, a violation of CP implies a violation of T, which could be observed by 
measuring the difference Py^^y^ = Py^^^^ for a ^ (5. 

The fact that the matrix Aw has only one non-zero element (Aiy)ii = Acc implies 
that 



V23 Aw V2Z — Aw ■ 

Therefore, it is convenient to define the new column matrix of amplitudes 

5(")(p,t)^4vl/(^)(p,t), 
which satisfy the simplified evolution equation 

z-^v^(°)(p,t) = — vl/(")(p,t), 
at 2p 

with 



(2.24) 



(2.25) 



(2.26) 



= Vn W12 Vl^ + Aw ■ 



(2.27) 
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Notice that the amphtudes in the column matrix '^^°'\p,t) do not have a definite fiavor 
or mass character. They are introduced only as a tool for the solution of the evolution 
equation ( |2.17| ). 

The standard procedure that has been used in finding the solution of Eq .( |2.26D consists 
of the diagonalization of the effective squared- mass matrix (see Refs.|E3, K^). This 



method is appropriate for a constant matter density or for a medium whose density 
changes along the path of neutrino propagation are much slower than the changes of the 
oscillation phases (adiabatic approximation). Techniques have been developed in order 
to extend the solution to non-adiabatic cases in which the matter density have a smooth 



and monotonic variation (as is the case for solar neutrinos; see Refs.[^, 10 ). However, 
the general case of irregular matter variations must be solved by integrating numerically 
Eq.(|;26D (or directly Eq.(^)). 

In the following we consider neutrino oscillations in the Earth, whose internal composi- 



tion is well approximated by a number of shells with constant density (see Refs.[39 



The solid curve in Fig.|I|A shows the density p in the interior of the Earth as a func- 
tion of the radial distance r from the center of the Earth according to the data 
given in Ref . ||7(J|| . The dotted curve represents our approximation in terms of five 
shells with constant densities pj=i,...,5 = 13.0, 11.3, 5.0, 3.9, 3.0 g/cm^, with outer radii 
ri=i,...,5 = 1221, 3480, 5701, 5971, 6371 Km. The solid curve in Fig.|B shows the electron 
number density iVg = p{Z/A) as a function of the radial distance r and the dotted curve 
represents our approximation in terms of five shells with constant electron number den- 
sity (A^e)i=i,...,5 = 6.15,5.36,2.47, 1.93, 1.50 A^Acm-3. For the averaged ratio {Z/A) we 
took {Z/A)i^2 = 0.475 for the two inner shells (core) and {Z/A)^^^^^ = 0.495 for the three 
outer shells (mantle). 

The effective squared-mass matrix can be diagonalized separately in each shell, 
yielding a relatively simple solution for the evolution equation ( |2.26| ) in the shell under 
consideration. The solutions in confining shells are matched on the shell boundaries by 
the continuity of the fiavor states, which imply the continuity of the amphtudes \l/[^^(p, t) 
and, from Eq.( |2.25D , of the amplitudes '^^°'\p,t) (the transformation ( |2.25| ) does not 
depend on the matter density). 

In the following we consider a scheme with the hierarchy ( |1.1| ) for the neutrino masses. 
In this case the oscillations of atmospheric neutrinos depend only on one mass scale. This 
scheme allows a relatively simple diagonalization of the 3x3 effective squared-mass matrix 
M2. 



2.1 One mass scale 



rrio 



rrin 



We will consider here a scheme with three neutrinos and the hierarchy ( |1.1|) for the 
neutrino masses. We will assume that the squared-mass difference Am 
is relevant for the explanation of the solar neutrino problem (Am: 
MSW resonant transitions or Am2i ~ 10 



^21 



10-5 eV^ ra for 



-10 , 



eV for vacuum oscillations). In this 
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case 



ml Rq 
2p 



< 1 



mi Rq 



2p 



< 1 



(2.28) 



where R^ = 6371 Km is the radius of the Earth, which represents a characteristic distance 
of propagation for atmospheric neutrinos. Hence, the phases generated by ml and m^ 
can be neglected for atmospheric neutrinos and the squared-mass matrix can be 
approximated with 



= diag(0,0,m^) . 



(2.29) 



Before proceeding with the derivation of the oscillation probabilities that follow from this 
approximation, it is necessary to notice that caution is needed for low-energy atmospheric 
neutrinos if m2 > 10^^ eV^, because in this case m|i?0/2p <^ 1 only for p ^ 150 MeV. 
According to the most recent analyses of the solar neutrino data [^, including prelimi- 
nary data from SuperKamiokande the small and large mixing angle MSW solutions 



of the solar neutrino problem require, respectively, 4 x lO'^eV^ < ^1 < 1-2 x lO'^eV^ 
and 9 x 10"^ eV^ < < 3 x 10"^ eV^ at 90% CL. Hence, expecially in the case of the 
large mixing angle MSW solution of the solar neutrino problem, the formalism described 
in this section may be not applicable to low-energy atmospheric neutrinos (as, for exam- 
ple, some part of those contributing to the Kamiokande and SuperKamiokande sub-GeV 
data). Since, as will be discussed later, the formalism described in this section is very 
convenient for the analysis of atmospheric neutrino data in the case of three-neutrino 
mixing (the oscillation probabilities are independent from 't}i2 and 7712), we think that if 
a high value of will be established, it will be convenient to analyze the atmospheric 
neutrino data with a cut in energy such that m^Ri^/^p <^ 1 (at least for a first analysis, 
before the calculation of a complete fit which should include also the solar neutrino data 
for the determination of m\ and 'di2)- Let us emphasize that there are good hopes that 
the value of m\ ~ Amgi will be determined by the new generation of solar neutrino 
experiments (SuperKamiokande, SNO, ICARUS, Borexino, GNO and others [^) that 
are expected to be able to distinguish among the different solutions of the solar neutrino 
problem. 

The simple form of the squared-mass matrix ( p^.29D allows one to write the oscillation 
probabilities in vacuum in an elegant way: using the unitarity relation 



Saf3 - Ub?.U, 



(2.30) 



k=l,2 



the amplitudes ( p.l5| ) can be written as 



.k=l,2 J ^ 



ml t 
~2p' 



q3 ^/33 



exp 



m\ t 



(2.31) 



10 



For the oscillation probabilities we obtain |7^, 

4p 



P,^^,, = A^,p sin^ ( ) {(3^ a), (2.32a) 



= 1 - E ^'^'^-'^/^ = 1 - ^i^' (l^) ' (2.32b) 



where 



^a;/3 = 4|[/„3nf//33|', (2.33a) 
Pa;a = = 4|f/,3|' (l - WaZ^) (2.33b) 

are the classical oscillation amplitudes. 

It is important to notice the following features of the oscillation probabilities (p.32|) : 

1. All oscillation channels (z/^ ^ z/g, ^ i^r, Ve ^ ^r) are open and have the same 
oscillation length 

Lose = ^ . (2.34) 
ml 

2. The transition probabilities are determined by three parameters: 777.3, l^esP and |f//i3|^ 
(from the unitarity of the mixing matrix it follows that |t/r3p = l-lf^esp-lf^/xsH- The 

I 9 I 2 

expression of \Uez\ and \U^2,\ in terms of the mixing angles of the parameterization 
(EUD is 

If/esP = sin2^9i3, |[/^3|2 = cos2^9,3 siu^ 7^23 • (2.35) 

Hence, the oscillation probabihties depend on the two mixing angles 7^13 and 7?23 and 
do not depend on the value of the mixing angle 79i2 and of the Dirac CP violating phase 
7^12. Since this fact is due to the degeneracy of the first two squared mass eigenvalues 
in Eg. ( p. 29)) , it remains valid for the oscillations in matter (see Eg. ( |2.49|) ). 

3. The expressions (lO^ ) have the same form as the standard expressions for the os- 
cillation probabilities in the case of mixing between two massive neutrino fields (see 
Refs.igi0,|l): 

(Am^ t\ 
-j^j iPi^a), (2.36a) 

P.^-^u^ = 1 - sin' 27? sin' (^^) ' (2-36b) 

where Am^ is the neutrino mass-sguared difference and is the two-generation mixing 
angle. In the case of two-neutrino mixing only oscillations between two flavors are 



11 



possible and the oscillation probabilities are characterized by two parameters, Am^ 
(which determines the oscillation length Lose = 47rp/Am^) and sin^ 2'd. The important 
difference between the two-neutrino mixing scheme and the scheme with three-neutrino 
mixing and a mass hierarchy that we consider here is that the second scheme allows 
simultaneous transitions among all three ffavor neutrinos (z/„ 



4. The equality in the form of the oscillation probabilities in the two-neutrino mixing 
scheme and in the three-neutrino mixing scheme (|1 . 1| ) is very important, because 
the data of all terrestrial oscillation experiments with reactor and accelerator neu- 
trinos have been analyzed by the experimental collaborations under the assumption 
of two-generation mixing, obtaining constraints on the possible values of the mixing 
parameters Am^ and sin^ Id. The results of the analysis of the experimental data 
are presented in the form of allowed (or excluded) regions in the sin^ I'd-lS.rv? plane. 
Identifying the appropriate A^-^p or B^-a with sin^ I'd and m\ with Am^, the results 
of the standard analyses of the neutrino oscillation data yield allowed (or excluded) 
regions in the A^.^j-mg and Ba-fx-m\ planes (with = e,fi,T). 



5. Since the classical oscillation amplitudes ( p.33| ) depend only on the squared moduli of 
the elements of the mixing matrix and Ajj-a = Aa-jj, it is clear that 



P. 



p 



P = P 



(2.37a) 
(2.37b) 



Hence, if neutrino oscillations depend only on one mass scale, CP and T violations are 
not observable. This is true for neutrino oscillations in vacuum as well as in matter, 
although in matter Eq.( p.37aD is no longer satisfied (see Eg. (|2.38| ) and the following 
discussion) . 

Let us now consider oscillations in matter and discuss how to solve the evolution 
equation (|2.26|) . The form (|2.29| ) for the mass matrix implies that Wi2M'^Wl2 = and 
the evolution equation ( p.l7| ) does not depend on the value of the mixing angle i)i2 and 
of the Dirac CP violating phase 1712. Then, from Eq. (|2.27|) we have 



= Vis V^^ + A 



w 



ml sin^ i!) 13 + Acc m| cos-i^is sint^ig 





y ml cosi^ig sint^ig 



13 



(2.38) 



The disappearance of the Dirac CP violating phase 1712 from the evolution equation for 
the transition amplitudes implies that, in the scheme under consideration, CP violation is 
not observable in neutrino oscillations in matter, as well as in vacuum. Let us emphasize, 
however, that, contrary to the vacuum case, in matter the survival and transition prob- 
abilities of neutrinos and anti-neutrinos can be different, because the effective potentials 
of neutrinos and anti-neutrinos have the same absolute value but opposite signs. Hence, 
Eq.( p.37aD is not satisfied in matter. This is due to the fact that the medium is not CP 
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invariant and not CPT invariant. However, if the matter distribution is symmetric along 
the neutrino path, the matter effect is T invariant and Pua^u0 = Pup-^ua matter as in 
vacuum (see Eg. ( p.37b|) ). 

Let us now proceed to diagonahze the matrix in Eq.( p.3^ ). Obviously, the matrix 
has a zero eigenvalue, 171^^2 = 0- The other two eigenvalues are given by 



"^1/1,3 = 2 ("^3 + ^cc) T-\J {ml cos 2??i3 - Acc) + {ml sin 2§i^y 



(2.39) 



These are the effective squared-masses of neutrinos propagating in matter. Notice that 
the values of the effective squared-masses of neutrinos and anti-neutrinos, which we will 
denote different, because Acc — ~^cc- 

Let us define the column matrix of amplitudes 



\^Ml{P^i)J 

where the orthogonal matrix 



Mt,T,(a), 



(2.40) 



*^13 — 



v 



COS<;^ 



sin<^ 



sin^9*| 



cos^9*^ 



13 / 



(2.41) 



is defined in such a way that the matrix 



Fif ^ V^^ ^ Ml ^ diag(m2,„ 0, m^s) (2-42) 
is diagonal. Then, we obtain that the effective mixing angle in matter is given by 

,2 o„Q ^ 



cos2t?f^ 



r/ig COS 2^9 



13 



ml cos2'i9i3 — Acc) + {ml sin2't?i3)' 



(2.43) 



If cos2'i9i3 > 0, for Acc ^ ml the effective mixing angle for neutrinos is approxi- 
mately equal to 7r/2. The corresponding effective mixing angle for anti- neutrinos, which 
we will denote as "d^^, tends to vanish because Acc is negative. When Acc = ml cos 21913 
there is a resonance: = 7r/4 and the mixing in the 1-3 sector is maximal. If 

cos2'(9i3 < 0, the resonance condition can be realized for anti- neutrinos {-^f^ = 7r/4) 



at Acc = cos2^9i3. In this case "i^ff ~ 7r/2 and 



M 



for Acc ^ ml. However, 



the analogy with the quark sector, in which there is a hierarchy of mixing that respects 
the mass hierarchy, suggests that a small value of is natural and that the resonance 
condition can be realized for neutrinos. 
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The evolution equation for t) is diagonal: multiplying Eg. ( p.26|) by on 

the left , we have 



M 



M ,T,(a) 



2p 



(2.44) 



with = diag(m|j^, 0, ml^g). It is important to notice that the assumption of a 
constant matter density plays here a crucial role. Indeed, if the matter density were 
not constant, the effective mixing matrix V^^ would be dependent on t and the time 
derivative of V^^ would induce additional non-diagonal terms in the evolution equation 

The exphcit evolution equation for the amplitudes t), with k = 1,2,3, is 



Mk J (a) 



(2.45) 



This equation has the straightforward solution 



m 



Mk 



2p 



(t - to) 



V^mIIp, ^0) 



(2.46) 



which can be written in a matrix form as 



exp 



■Ml , 



2p 



*l^(P,to) 



(2.47) 



This equation gives the evolution in time of the amplitudes ilj^^\{p,t). However, we 
are interested in the evolution in time of the amplitudes {p, t) whose absolute value 
squared give the oscillation probabilities trough Eq.( |2.14| ). The amplitudes ip^f^\p,t) are 
the elements of '^^\p,t) and, using Eqs.( p^.25D and ( |2.4(J| ) we can express '$^\p,t) in 
terms of \E'^^ (p, t) as 



(P, t) = (p, t) = (p, t) = ¥^ {p, t) 



rM ,Tf(")/ 



(2.48) 



with the orthogonal matrix 



sin ^^23 sin 
cos '^23 sin 



M 





COS ^^23 

- sin -(923 



sin t?]^ 
sin '(923 cos 



cos '621, cos -i^^f y 



(2.49) 



Notice that the elements of the third column of V^^ have the same structure as the 
elements of the third column of the mixing matrix U (see Eq. ([A.20| )), in terms of the 
mixing angle 'i923 and the effective mixing angle in matter -(9*1, which replaces the vacuum 
mixing angle "(^is. Let us emphasize, however, that the matrix V^'^ can be considered as 
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the effective mixing matrix in matter only for the study of the oscillations in matter in the 
scheme with three neutrinos and the mass hierarchy ( |1.1|) . In this case, the approximation 
mi = 7712 = implies that there is no mixing in the 1-2 sector and 1^12 is equal to the 
identity matrix. Notice that the first row of V^^ , corresponding to the electron neutrino, 
is particularly simple, with Vj^' = 0, because the electron neutrino feels only the mixing 
in the 1-3 sector. Physically, the special status of the electron neutrino follows from its 
direct charged current interaction with the electrons in the medium, represented by the 
quantity Acc- 

Substituting the solution (|2.47|) for \E'j^'' (p, t) and using the orthogonality of V^'^ 
obtain 



we 



exp 



Ml 
2p 



{t - to) 



M 



{P, to) 



Explicitly, we have 



^fip,t) = J2^M^^p 



k,p 



m 



Mk 



2p 



(t - to) 



(a) 



{P, to) 



(2.50) 



(2.51) 



This is the solution of the evolution equation for the amplitudes ip^^^ (p, t) in a slab of 
matter with a constant density. In the case of neutrino propagation through a series of 
slabs of matter with a constant density, as in the interior of the Earth, the solutions ( |2.51| ) 
for confining slabs must be matched in order to have continuity of the flavor amplitudes 
ip^^\p,t). For example, for a neutrino created at the time to and crossing a series of 
boundaries between slabs of matter with a constant density at the times ti, t2, ... , tn, 
the amplitudes 'ipl^\p,t) in the (n + 1)**^ slab are given by the matrix equation 



^^w\p^t) 



M -i 



2p 



M\T 



(n+1) 



M -i 



.M 2^(t„-t„_l) 



2p 



M\T 



(n) 



M-i 



2p 



M\T 



J (2) L 



. *fL (ti -*n) 



^P{p,to). 



(1) 



(2.52) 



Here the notation [. indicates that the matter-dependent quantities inside of the 
square brackets must be evaluated with the matter density of the i^^ slab. From Eq.( p.l4] ), 
one can see that the oscillation probabilities are given by the squared moduli of the 
amplitudes ip^^\p,t) given by Eq.( |2.52| ). 

It is clear that for n > 1 an analytical calculation of the oscillation probabilities 
is a rather complicated task and of little interest. Therefore, this calculation is done 
numerically with a computer. 

The calculation for n = 1, i.e. for the case of a medium with constant density, can 
be done analytically. In order to derive the oscillation probabilities, we employ the same 
method as in vacuum (see Eqs.( p.30D -( p.33b|) ): using the orthogonality relation 



E 

fc=l,2 



yMyM 
^ 13k ^ak 



M-irM 
(33 ^a3 ' 



(2.53) 
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the amplitudes ( p.51j ), with to = and ip'f\p, 0) 



6ap, can be written as 



^al ^/31 



A/1 



2p 



, YM yM I T/M yM 



exp 



— z 



A/1 



2p 



exp 



a3 »//33 exp 
2 



"^A/3 ^ 



2p 



2p 



exp —t 



- 1 



(2.54) 



2p 



with 



^"^A/31 



A/3 



m 



Ml 



m| cos2'(9i3 — A. 



cc 



)' + K sin2i9i3)' 



For the transition and survival probabilities we obtain 



Pu 



aM c2 a t/A/ t/A/ t/A/ t/A/ c2 /I t/A/ t/A/ t/A/ t/A/ / c2 c2 \ 
^q;/3'J31 ~ ^ '^al ^ pi ^ a2 ^132 "^l " ^ »^q2 ^132 ^a3 ^(33 ["^S ~ '^31 j 

Si - 4 (rif )^ (rl)^ - 4 {V^'if iV^lf {Si - , 



1 



respectively. Here 

02 -2 

= sm 

and 



Ap 



r,2 
•^3 



sm 



^A/3^ 

4p 



r.2 
^31 



sm 



^"^A/31^ 

Ap 



B. 



M 
a;f3 

M 



o3 
f3^a 



(2.55) 

(2.56a) 
(2.56b) 

(2.57) 



(2.58a) 
(2.58b) 



are the classical oscillation amplitudes in matter analogous to the corresponding ones in 
vacuum, A^-p and Ba-^ (see Eqs.( |2.33| )). The classical oscillation amplitudes in matter 
A^.f^ and B^.^ have the same structure as the classical oscillation amplitudes in vacuum 
Aa-^j3 and Ba-a, in terms of the mixing angle and the effective mixing angle in matter 
which replaces the vacuum mixing angle di:^. Therefore, it is easy to see that in 
the limit Vcc only the terms involving A^.^ — >• A^-p and B^.^ Ba-a survive 
in Eqs. (|2.56|) and the oscillation probabilities reduce to the oscillation probabilities in 
vacuum ( ^.321 ). 

It is interesting to notice the following features of the oscillation probabilities (|2.56| ): 

1. The oscillation probabilities involving the electron neutrino in the initial and/or final 
state are particularly simple, because = 0: 



P 



T^M q2 
-°e:e ^31 ; 



P. 



A A/ c2 
^e;n ^31 5 



(2.59) 



with 



B^i 



sin^ 2^ 



M 
13 ! 



A^=sm'^23 sm'2^^i. 



A/ 



(2.60) 



16 



These oscillation probabilities depend only on one oscillation length, 

iS'^^. (2.61) 

(31) 

In the limit Vcc — > we have Lose —>■ Lose, where Lose is the oscillation length in 



vacuum, given in Eg. ( p.34 ) 



2. The oscillation probabilities involving only muon and/or tau neutrinos have a compli- 

•(1) 

^ose; 



cated structure, depending on three different oscillation lengths: L^sc'*, L^sc and Lo^-* 



with 

-^osc ~ 2 ' -^osc ~ 2 ■ (2.62) 

In the limit Vcc ^ 0, we have that Lisc"* — > Lose, -^osc Lose and lIsc — ^ oo. 
The survival probability of muon neutrinos is given by 

P,^^,^ = 1 - P;f^ si - sin' 2^23 sin^ S', - sin' 2^23 cos' ^fi {S', - S^) , (2.63) 

with 

= sin^ sin' 2^^^ + sin' 2^23 cos' . (2.64) 



3. The oscillation probabilities ( 2.56|) are invariant under time reversal: Py^^y^ = Pi 



This is due to the fact that the effect of a constant matter density along the neutrino 
path is T invariant. As a consequence, all the oscillation probabilities can be derived 
from the three probabilities (|2.59|) and (|2.63|) using the conservation of probabilities 



i^pPva^vi3 = 1) and the invariance under time reversal (Pi^^^i,^ = Pj^i^^y^). 

4. The survival and transition probabilities of neutrinos and anti-neutrinos can be dif- 
ferent, because the effective mixing angles '(?f| and the effective squared-masses 
"^Mi3' ^\n3 neutrinos and anti-neutrinos are different {Acc = —Acc)- This is a 
consequence of the fact that the medium is neither CP invariant nor CPT invariant. 

When Acc ^ which can be realized with high matter density and neutrino en- 
ergy, the effective mixing angle of neutrinos becomes large and approaches 7r/2 for 
Acc ^ "^3 (see Eq. ( p.43|) ; here we assume cos2't?i3 > 0). This means that the matter ef- 
fect is relevant and can dramatically modify the oscillation probabilities. From Eqs.( p.59 ) 



and ( |2.6CI| ) we see that, in the case of a medium with constant density, when Acc ^ "^3 



and ^ 7r/2 the transition probabilities of electron neutrinos are suppressed. The same 
suppression is realized for anti-neutrinos, which have d\l ~ 0. Assuming a small value for 
'(?i3, the maximum difference between the neutrino and anti-neutrino oscillation probabili- 
ties is obtained when the resonance condition Acc = ""^3 cos2'(9i3 is satisfied for neutrinos, 
which implies that i)^ ~ 7r/4 for neutrinos, whereas < i^i^, for anti- neutrinos. At the 
resonance the oscillations of the survival probability of electron neutrinos are maximal, 
Puc^ve = 1 ~ "S"!^, while the other probabilities depend on the value of ^923- 
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2.2 Low energy neutrinos 

It has been pointed out Q that the matter effect may be not neghgible even in the 
case of low matter density or low neutrino energy^, i.e. for Acc = SpVcc ^l- In this 
case, the mass eigenvalues can be approximated by 

m^i ~ Acc(l + cos2^9i3)/2, (2.65a) 
ml,^ ~ + Acc (1 - cos 2^r3) /2 , (2.65b) 
Am^^3^ ~ m3 — Acc cos2'(9i3 . (2.65c) 

It is important to notice |j76| that m\j^ is proportional to Acc, which in turn is propor- 
tional to the neutrino energy {Acc = '^pVcc)- Hence, the phase 

^ — (1 + cos 2i9i3) = — — (1 + cos 2i9i3) (2.66) 

2p 4p 2 

is independent of the neutrino momentum and can be relevant for low-energy atmospheric 
neutrinos (as those corresponding to the Kamiokande sub-GeV data). The value of the 
potential Vcc is given by 

Vcc = V2GfN, = 7.63 x IQ-^^ { 

/ / (2-67) 

where Na is the Avogadro number. The interior of the Earth has an electron number 
density Ne that goes from about 2 A^^ cm~^ in the mantle to about 6 Na cm~^ in the 
inner core, with an average value of about 3 Na cm~^ (see Fig.|l]B) . For a propagation of 
10^ Km we have Vcc'l^/'^ — 2vr, which shows that the phase ( p.66| ) could be relevant for 
low-energy atmospheric neutrinos. 

When Acc rrig, the oscillation probabilities depend on two oscillation lengths: 

^gc^ ^ ^i'i ^ ^ = LtT , ^ , ^ L^-^ . (2.68) 

"'"^ ml "'"^ Vbc (1 + cos2?9i3) ^ ' 



The short oscillation length L^^™^ coincides with the oscillation length in vacuum, whereas 
the long oscillation length L^^sc" is due to the matter effect. For example, for m| = 
10~^eV^ (which is close to the best fit of the atmospheric neutrino data), p = 500 MeV 
(which is in the range of the Kamiokande sub-GeV data), A^e = 3A^cm~'^, we have 
L^^°'^ ^ 10^ Km and Lj,"^^? ~ 10^ Km. The oscillating terms sj and sl^ in Eq.(g37D 



Let us remind that, as discussed after Eg. ( 2. 29 ), caution is needed for low-energy neutrinos if 
> 10-5 eV^ In this case the considerations presented in this subsection can be apphed only to 
neutrinos in the momentum range m|i?®/2 <C p <C ml/2Vcc- For example, for = 1.5 x IQ-^ eV^ 
(that corresponds to the best-fit point of the large mixing angle MSW solution of the solar neutrino 
problem |0), ml — 10"^ eV^ and Vcc = 5 x 10""'^^ eV (that corresponds to the density in the core of 
the earth) we have the momentum range 200 MeV <C p <C 10 GeV. 
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depend on the short oscillation length Losc"^* oscillate very fast on a distance scale 
bigger than 10'^ Km. Therefore, one can study the slow oscillations of the probabilities 
due to L^osc^, averaging the probabilities over the fast oscillations due to I^og™*: 

{Pu.^uJ =1-1 C , {Pu.^u,.} = \ 41 , (2-69a) 



<^..^..> = 1 - \b^, - sin^ 2^23 sin^ sin^ 



rlong 
J^osc 



(2.69b) 



with and Bf^^^ given by Eqs.(p:BU[) and (^) 



One can see that the averaged value of the survival and transition probabilities of 
electron neutrinos are constant and very close to the corresponding averaged probabilities 
in vacuum (for Acc -C m| we have dfi^ ~ -i^ia, B^^ ~ B^-e and ~ ^e;/^), 

{Pu,^Ue)viK ^ ^ ~ 2 ^'"''^ ' (^"'^^''f^) vax: ^ 2 "^""'^ ' (2.70) 

On the other hand, the averaged value of the survival probability of muon neutrinos 
has the additional term depending on L^°^^ with respect to the corresponding averaged 
probability in vacuum 

(^^.-.Lc = 1-^^m;m- (2-71) 

The behaviour of (^Py^^^^^ as a function of the propagation distance L = t in a medium 
with a constant electron density A^^g = 3 Na cm~^ is depicted in Fig.^ for neutrinos and 
anti- neutrinos and two sets of values of the mixing parameters: (A) |t/e3p = If^A^sl^ = 1/3 
(which corresponds to maximal mixing of the three neutrinos) and (B) |t/e3p = 0.3, 
|t^/i3p = 0.5 (which is close to the best fit of the Kamiokande data; see Eq.( |3.6|) ). The 
solid lines represent (Pj^^_,y^), the dashed lines represent (Pp^^p^) and the dotted lines 
represent the averaged survival probability in vacuum ( p.71| ). From the comparison with 
the averaged survival probability in vacuum, it is clear that the oscillations in matter can 
be rather different than in vacuum even if Acc One can also see a small difference 

between the averaged survival probabilities of muon neutrinos and anti-neutrinos, due to 
the fact that and t^ff are not exactly equal to "t^i^ and different between each other. 

In Fig.^,B we present the behaviour of i^Py^^y^^ and (Pp^^p^) for neutrinos prop- 
agating in the interior of the Earth as a function of the zenithal angle 9. These figures 
correspond to the same two sets of values of the mixing parameters as in Fig.|[ The val- 
ues of (^Pi,^^y^^ (and similarly those of (Pp^^p^)) are obtained assuming m\ = 10~^eV^, 

p = 500 MeV and averaging numerically the probability P^^^-^^ = ipji" {p, t) , with 

ilj'^\p,t) given by Eq. (|2.52| ), over the fast oscillations corresponding to i^osc"^* — 10^ Km. 
Figure shows the slant depth as a function of the zenithal angle 6. One can see that 
the irregularities of (Pj/^^z/^) and ^Pp^_^p^) correspond to irregularities of the slant depth, 
which occur for trajectories that graze the boundaries between the different shells with 
approximately constant density. 
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The effect of L^°^^ will be negligible in the long-baseline experiments of the next gen- 

, KEK-SuperKamiokande |3^, Fermilab-Soudan 
^ ^ [35]) because the baseline will be shorter than 10'^ Km. How- 

ever, it is interesting to notice that L^°^^ is independent from the value of and is 
very sensitive to the matter density. Therefore, the observation of the effect of L^^sc^ in 
future very-long-baseline neutrino oscillation experiments could be helpful for a study of 
a detailed tomography of the interior of the Earth |l78|. 



3 Analysis of the Kamiokande atmospheric neutrino 
data 

In this Section we present our fit of the Kamiokande atmospheric neutrino data in the 
scheme with mixing of three neutrinos and a mass hierarchy considered in the previous 
Section. In this scheme the probability P^^^^^{E,cos9;ml,\Ue3\'^,\U^3\'^) of 
oscillations for a neutrino with energy E c:^ p arriving at the detector from a direction 
with a zenith angle 6 depends on the three parameters m|, |?7e3p, I^a^sP matter 
density along the neutrino trajectory. (In this article, we do not consider preliminary data 
from the SuperKamiokande experiment, waiting for more refined data to be available in 
the future.) 

3.1 The experimental data 

We have analyzed the neutrino-induced e-like and /x-like event rates measured by the 
Kamiokande [0, 0, ^] experiment. 

The Kamiokande data sample is divided into two classes: the low energy sub-GeV 
data, including only fully contained events, and the high energy multi-GeV data, including 
both fully and partially contained events. Additional information on these events is 
provided by their angular distributions, divided in five zenith-angle bins. In our analysis 
we fit the angular distribution of the e-like and //-like multi-GeV data and the total 
number of e-like and /i-like sub-GeV data reported in Ref.0. For the sub-GeV data 
we do not fit the angular distribution because there is a poor correlation between the 
directions of the neutrino and the produced charged lepton 0. 

The Kamiokande collaboration measured the absolute rate of e-like and /i-like events. 
In order to analyze the muon-electron flavor composition of atmospheric neutrino events 
they presented also the ratio of ratios R^/e = (/i/e)exp/(yu/e)th, where (/i/e)exp is the 
ratio of yU-like and e-like events measured experimentally and (yu/e)th is the same ratio 
calculated theoretically, without neutrino oscillations. The Kamiokande results 

^sub-GeV^Q_gQ+0.06^Q_Q5^ (3-1) 

^)ve"'"^'^ = 0.57+°:°? ± 0.07 (3.2) 

indicate an anomalous flavor composition in the observed atmospheric neutrino flux, in 
both the sub-GeV and multi-GeV energy ranges. 
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In our analysis we did not use the ratio of ratios R^/e, because, according to the 



arguments discussed in Ref.|73], its non-Gaussian probabihty distribution might bias the 
statistical analysis. Therefore, we preferred to study the separate e-like and /x-like event 
rates, including in our analysis the correlation of their uncertainties. 

3.2 Theoretical calculation 

The expected numbers of e-like and /i-like events in each angular bin, under the hypothesis 
of neutrino oscillations, are given by 

Ne^ = Nl^ Pi:L,^ + N'Jl Pi;L,^ , (3.3) 
N,^ = P^^^^ + iV;^ P»_^,^ , (3.4) 

where and A^*^ are, respectively, the number of e-like and /i-like events in the angular 
bin i calculated theoretically, under the assumption that there are no oscillations, while 
Pu}~*vi3 is the averaged probability of vp transitions (with a, (3 = e, /i) in the 

bin i. The five bins of cos 6' for the Kamiokande multi-GeV data are centered around 
(cos 6')^^-,^ 5 = —0.8, —0.4, 0.0, 0.4, 0.8, which correspond to the average distances 
{^)i=i 5 — 10230, 5157, 852, 54, 26 Km travelled by neutrinos from the production 
point down to the detector site. 

Since the neutrino oscillation probabilities Py^^^^ are functions of the neutrino energy 
and of the zenith angle, in order to compare the theoretical calculation with the experi- 
mental data, they must be averaged both on the neutrino energy spectrum (different for 
e-like and /x-like events) and on the angular width of each zenith-angle bin used in our 
analysis. 

Therefore, a preliminary step in our calculations consisted of the evaluation of the 
averaged probabilities Py}^yij{m\, jf/esP, I^msH ^c^ch. angular bin i, for a fixed set of 
values of the three oscillation parameters m|, |?7e3p, It^^sP' defined by the double-integrals 

PS^,^{ml\U,,\\\U,,\'') = j dcos^ j dES,^{E)Py^^y^{E,cose-ml\U,,\\\U,,\''), 

(3.5) 

where 6 is the zenith angle, E is the neutrino energy and Sy^{E) is the energy spectrum 
of the parent neutrino (with flavor a). These energy spectra were extracted from the 
figures published in the Ref.p for the Kamiokande sub-GeV data and in Ref.fQ for the 
Kamiokande multi-GeV data. 

When matter effects are included in the calculations, the probabilities 
Py^^y^(i?, cos 6*; mg, |[/e3p) |f^M3p) ^'^^ simple functions of the neutrino energy and 
zenith-angle, but must be calculated numerically by taking into account the matter den- 
sity along the neutrino path in the Earth. The density and composition of the interior 
of the Earth is known from seismological measurements (see Refs. pU|, |70[). The electron 



number density N^. varies in a discontinuous way from 1.6A^^cm~^ near the surface to 
6.2A^yicm~^ in the inner core. However, it is possible to approximate the electron num- 
ber density profile with a step-like function, each step representing a shell with constant 
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density (see also Refs.|]23|). We approximated the numerical data in Ref.|7^ with five 
shells of constant density (see Fig.|I]) and we considered the atmosphere as an outer shell 
with a thickness of 20 Km. The effective neutrino path in each of the shells depends upon 
the zenith angle of the neutrino trajectory. 

The amplitude of Pa transitions for a neutrino with energy ^ p in a medium 

with constant density is given by Eg. ( |2.51| ). In the case of a neutrino crossing various 
shells of different density, the matrices describing the propagation across different shells 
must be multiplied serially, as in Eq.( p.52 ), in order to get the amplitude of the transition 



along the total path. The square modulus of this amplitude gives the probability of 
— ^ ^13 oscillations (see Eq. (|2.14|) ). This probability must be folded with the relevant 
neutrino energy spectrum and integrated over the selected intervals of energy and cos 6 
(see Eq.(U)). 

We took the values of iV*]^ and Nj^ given by Table 2 (flux A) of Ref.|| for the 
Kamiokande sub-GeV data and by Fig.3 of Ref. for the Kamiokande multi-GeV data. 
The estimated uncertainties of the absolute value of N^f and A''*^ amount to 30%, mainly 
due to the uncertainties of the value of the calculated electron and muon neutrino fluxes. 
The estimated relative errors for the ratios Nj^/N^f ratio are much smaller: 9% for 
the Kamiokande sub-GeV data and 12% for the Kamiokande multi-GeV data. In our 
calculations we neglected the fact that not all e-like and /i-like events are produced by 
Ue {i>e) and z/^ (i/^) interactions, respectively. The purity of the e-like and /x-like events 
is estimated by the Kamiokande Collaboration to be higher than 90% [Q. 

3.3 Results of the fit 



The number of e-like and /x-like events was calculated using the formulas (|3.3|) and (|3.4| 



for a grid of values of the parameters ml, |f/e3p, I^^msI^' averaging over the contributions 
of neutrinos and anti-neutrinos. These values were used together with the experimental 
results to build a proper function. The best flt of the Kamiokande data was obtained 
for 

= 2.5 X 10-2 eV\ pes]'^ = 0.26 , l^/^gj^ = 0.49 , (3.6) 

with = 6.9 for 9 degrees of freedom, corresponding to a CL of 65%. 

The regions allowed at 90% CL by the Kamiokande data in the Be-e-ml, B^.^^-m\, 
Af^-e^ml and A^..r-m\ planes are presented in Figs.^, |], ^ and ^ respectively (shadowed 
regions). Since the oscillation probabilities in vacuum ( ^.321 ) have the same form as the 
corresponding ones in the case of two-neutrino mixing (see Eqs. (|2.3^ )), the allowed re- 
gions in the Be-e^ml, B^-^-m\, A^-e^ml and Af^-r^ml planes can be compared directly 
with the exclusion plots of the terrestrial reactor and accelerator neutrino oscillation 
experiments whose data have been analyzed by the experimental collaborations under 
the assumption of two-generation mixing. This is simply done through the identiflca- 
tion of the appropriate Aa^p or B^-a with sin^ 2{} and the identiflcation of Am^ with 
m|. Moreover, the expected sensitivities of future long-baseline experiments are also pre- 
sented by the experimental collaborations as curves in the sin^ 2i)-Am? plane relative 
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to two-neutrino oscillations in vacuum. Therefore, each of these sensitivity curves can 
be compared directly with the corresponding allowed region in one of the Aa-^p-m\ or 
Bct-a^Tn\ planes. The same reasoning applies to the future results of reactor long-baseline 
experiments ||3l|, whose neutrino beams propagate in vacuum. On the other hand, 



the neutrino beams of accelerator long-baseline experiments [^, ^ |3^ will propagate in 
the Earth and matter effects may be significant. Since the effects of matter are different 
in the cases of two-neutrino and three-neutrino mixing, in order to get information on the 
mixing parameters in the scheme with three neutrinos with a mass hierarchy considered 
here it will be necessary to analize the data with the formalism presented in Section |^ 



The dashed and dotted curves in Fig.HI represent, respectively, the exclusion curves 



obtained in the Bugey and Krasnoyarsk ||6T[ reactor z/g disappearance experiment 
(the excluded region lie on the right of the curves). One can see that a part of the 
shadowed region allowed by the Kamiokande data in the Bf..e-m\ plane is excluded by 
the results of the reactor neutrino oscillation experiments (the lightly shadowed region). 
Only the darkly shadowed region in Fig.^ is allowed by the results of the Kamiokande and 
reactor experiments. In Fig.^ we have also plotted the sensitivity curves of the CHOOZ 
| 3T| and Palo Verde |0 long-baseline z/g disappearance experiments with reactor anti- 
neutrinos (the dash-dotted and dash-dot-dotted curves). One can see that a large part 
of the allowed region will be explored by the CHOOZ and Palo Verde experiments. 

The region allowed by the Kamiokande data in the B^-^-m\ plane, shown in Fig.^, 
is not constrained by the results of terrestrial neutrino oscillation experiments. Indeed, 
the best limit at small values of Am^ is provided by the CDHS disappearance 
experiment, whose exclusion curve lies rather far from the Kamiokande-allowed region, 
as shown in Fig.^ (the dashed curve). In Fig.^we have also plotted the sensitivity curves 



of the K2K |^3[ and MINOS long-baseline experiments (the dash-dotted and dash- 
dot-dotted curves), which show that the Kamiokande-allowed region will be explored in 
the near future. 

A part of the shadowed region allowed by the Kamiokande data in the A^.g-mg plane, 
shown in Fig.^, is excluded by the results of the reactor neutrino oscillation experiments 
(the lightly shadowed region), leaving the darkly shadowed region allowed by the results of 
the Kamiokande and reactor experiments. As shown in Fig.^ most of this allowed region 
will be explored by the K2K, MINOS and ICARUS ||35| long-baseline experiments, whose 
sensitivity curves are represented by the long-dashed, dash-dotted and dash-dot-dotted 
curves, respectively. 

Finally, Fig.|^ shows the Kamiokande-allowed region in the A^--r-m\ plane, part of 
which is indirectly excluded by the results of the reactor neutrino oscillation experiments 
(the lightly shadowed region). Indeed, in the lightly shadowed region the amphtude A^^.^^ 
is rather small and the fit of the Kamiokande data requires large values of the amplitude 
A^ -i^^, which, because of the inequality A^ .^^ < By^.y^, lie in the region excluded by 



^ Let us notice that, since the energy of accelerator neutrinos in long-baseUne experiments is of the 
order of 10 GeV, the formaUsm presented in Section ^ can be apphed without caveats also in the case of 
the large mixing angle solution [^8[ ^ of the solar neutrino problem, for which ml<ix IQ-^ eV^ and 
m^i?e/2p< 5 X 10-2. 
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the results of the Bugey and Krasnoyarsk reactor experiments. Hence, only the darkly 
shadowed region is allowed by the results of the Kamiokande and reactor experiments. 
As shown in Fig.|^ this region will be explored by the MINOS and ICARUS long-baseline 
experiments, whose sensitivity curves are represented by the dash-dotted and dash-dot- 
dotted curves, respectively. Since the allowed region extends at large values of A^^-^^, 
the Kamiokande data indicate that z/^ —>■ Uj. oscillations could be observed with a large 
statistics by the MINOS and ICARUS experiments. 



The results of our analysis are similar to those presented in Ref. [24|, where the same 



scheme with three neutrinos and the mass hierarchy ( p..l|) was used, but matter effects 
were not taken into account. However, one must notice that the presence of matter is 
important because it modifies the phases of neutrino oscillations ||75[ and its effect is to 
slightly enlarge the allowed region of the mixing parameters towards low values of m|. 



4 Conclusions 

We have presented a comprehensive formalism for the description of neutrino oscillations 
in the Earth in the case with three massive neutrinos whose masses satisfy the hierarchical 



pattern (|1.1| ). Such a scheme is allowed by the see-saw mechanism for the generation of 
neutrino masses and permits one to explain the solar and atmospheric neutrino problems 
through neutrino oscillations if m| ~ 10~^ or 10~^° eV^ and m| ~ 10~^eV^. 



In Section |2.2| we have discussed the matter effect on the oscillations of neutrinos with 
low energy and have shown that the oscillation probabilities depend on two oscillation 
lengths: L^^°^^ which is the same as the oscillation length in vacuum and L^°^^ ~ 10^ Km 



see Eg. ( [2.68 )). The oscillation length L^°^^ is independent of the value of the mass of 



the heavy neutrino and the neutrino energy, and is very sensitive to the matter density. 
Future very-long-baseline neutrino oscillation experiments could observe the oscillations 
due to i^osc^, after averaging the oscillation probabilities over the fast oscillations due 
to i^os™*- The sensitivity of these averaged oscillations to the matter density along the 
neutrino path could allow one to obtain a detailed tomography of the interior of the 
Earth. 

In Section |^ we discussed the analysis of the Kamiokande atmospheric data and pre- 
sented our results in the form of allowed regions in the Be-e-m^, B^.^^-m\, A^.^e^m\ and 
Afj^-T^ml planes (see Figs.|, |^, ^ and |^). The knowledge of the allowed regions in these 
planes is useful because the oscillation probabilities in vacuum ( |2.32| ) in the scheme under 
consideration have the same form as the oscillation probabilities in vacuum (|2.36| ) in the 



case of two-neutrino mixing, which have been used by the experimental collaborations in 
the analysis of the data of terrestrial oscillation experiments with reactor and accelerator 
neutrinos, yielding exclusion curves in the sin^ plane. Identifying the appropri- 

ate Aa-i3 or Ba-a with sin^ 2'd and m| with Am^, the exclusion curves in the sin^ 2'd-/S.rn? 
plane obtained by the terrestrial neutrino oscillation experiments can be plotted directly 
in the Bf,.e-m\, B^.^-ml, A^-e^ml and A^.T--m\ planes and can be compared directly with 
the regions allowed by the analysis of the Kamiokande data. The same reasoning applies 
to the sensitivity curves and future results of long-baseline experiments evaluated under 
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the assumption of two-neutrino oscillations in vacuum. 

Our analysis of the Kamiokande data took into account the presence of matter in the 
oscillations of neutrinos passing through the Earth, whose effect is to slightly enlarge the 
allowed region towards low values of mg. The best fit is obtained for ml = 2.5 x 10~^eV^ 
and from Figs.^, ^ |^ and |^ one can see that the allowed region of mg extends from about 
3.5 X 10-3 eV^ to about 3.5 x IQ-^eV^ 

From Figs.§, ^ ^ and |^ one can see that the long-baseline experiments with reactor 
(CHOOZ and Palo Verde) and accelerator (K2K, MINOS and ICARUS) neutrinos could 
observe neutrino oscillations in all channels with a relatively large statistics^. 

The region in the A^.^T--m\ plane allowed by the results of the Kamiokande and reactor 
neutrino oscillation experiments extends at large values of A^^.^^ (see Fig.|^). Hence, the 
Kamiokande data indicate that the MINOS and ICARUS long-baseline experiments could 
observe a relatively large signal in the channel — > z/^. This could be the first direct 
observation of the tau neutrino. 
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Section 2.2 



A The mixing matrix 

A 3 X 3 unitary matrix U can be written as (see Refs. PUI 

U = D{uo) WWabi^ahe''^''') (a, 6= 1,2,3), 



a<b 



with the unitary matrices 



D{u) = diag(e*'^^ , e^'^ , e*"^) , 

+ sin i^ab (e*''"" 5ra 5sb - e"'''"" 5rb 5sa) 



(A.l) 



(A.2) 
(A.3) 



^ After we finished this work the CHOOZ collaboration disclosed its first results (M. ApoUonio et ai, 
Phys. Lett. B 420, 397 (1998)), which are compatible with the hypothesis of absence of i>e oscillations and 
lead to the upper bound sin^ 2-3 < 0.18 for Am^ > 10"^ eV^ at 90% CL. This limit is close to the CHOOZ 
sensitivity curve shown in Fig.^ and implies that large i'^ ^ i^e oscillations of atmospheric neutrinos are 
excluded. However, a subdominant contribution from ^ v,, oscillations to the atmospheric neutrino 
anomaly remain s an interesting possibility (see also G.L. Fogli, E. Lisi, A. Marrone and D. Montanino, 
hep-ph/971142l| ). 
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Here D^u) is a diagonal matrix depending from the set of phases u = {u!i,u 
the matrices Wabi'^ab^^^"'') are unitary and unimodular. For example, we have 

^ cos 'di2 sin ■(?i2 e*''^^ 0^ 



u = (tui, UJ2, ujs) and 



W^i2(^9i2e 



The matrices -D(A) and Wa6(''?afee*'''"^) satisfy the useful identity 

). Using the identity ( |A.5| ) it is 



sin'i9i2e cos'i9i2 









(A.4) 



V 



for any set of phases A = (Ai, A2, A3). 
Wabi^ab^^^"'') can be written as 

Wabi^abe'"'^") =DabVabDl, 



(A.5) 

it is clear that the matrix 



(A.6) 



with 



[Dabls = Srs + {e'''^' - 1) Sra Ssa , (A.7) 
[Vablrs = ^rs + (cOS'dab " 1) (<5ra Ssa + ^rb ^sb) + slu {5ra Ssb " Srb Ssa) ■ (A.8) 

The matrix Vab operates a rotation of an angle i)ab in the Ua^i^b plane. For example, we 
have 

^ cost9i2 sin?9i2 o\ ( e''''^ o\ 



Vi2 = 



The expression ( |A.l 



— sin'(9i2 cos'(9i2 
for f/ can be written as 



1^12 = 



1 
\o OlJ 



(A.9) 



U = D{uj- A) [\{D{\)Wab{^abe'^^') DiXy D(A) 



(A.IO) 



\a<b 



with the set of arbitrary phases A = (Ai, A2, A3). Using the identity ( |A.5| ), we have 



U = D{u- A) ( n W^a6Kfce*^"''+^'^'-'''0 ) ■ 

\a<b / 



(A.ll) 



The set of arbitrary phases A = (Ai,A2,A3) can be chosen in order to extract two of 
the three phases rjab from the corresponding Wab- Only two phases rjab can be extracted 
because there are only two independent differences A^ — A^. For example, one can extract 
?7i3 and 7723 with the choice 



Ai — A3 



-^713 



Ao ~ Aq 



-^23 



(A.12) 
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Then, rju cannot be extracted because Ai — A2 = — "/^la + ^723 is no longer arbitrary. 

The order of the product of the unitary matrices Wab in Eq.( [A.l|) is arbitrary. Making 
the choice ( |A.12| ) and A3 = 0, the mixing matrix can be written as 



U = D{co- A) V23 Vn W12 D{\) . 



(A.13) 



with 



e 

-D12V12-D12 J 



t 



) 



(A.14) 
(A.15) 
(A.16) 



D(cu-A) = diag(e^('^^+''") 

D{X) = diag(e-*'?" , e-'"^^ , l) 

The mixing matrix U appears in the lepton charged-current (the charged lepton fields 
are defined as mass eigenstates, because they are observed in the experiments, whereas 
neutrinos are never observed directly) 

2 ^ ^ Lil^U^k^kL 

a=e,fi,T fc=l,2,3 

a,k 



a=e,fj,,T 



(A.17) 



where £aL represents the left-handed component of the charged lepton field ia (with 
a = e,fi,T). The three phases in D{uj — A) can be eliminated with a redefinition of the 
arbitrary phases of the charged lepton fields, ia ia^~^^^°'~^°'\ which leads to 



2 ^'^ [^23 V^13 W12 D{\)]^^ UkL 



(A.18) 



If neutrinos are Dirac particles, also the phases of the neutrino fields are arbitrary and 
the two phases in -D(A) can be eliminated with the redefinition z/^ — > e"*^'' z/^. In the 
case of Majorana neutrinos the elimination of the two phases contained in -D(A) (called 
in this case "Majorana phases") is not possible, because the Majorana mass term is not 
invariant under rephasing of the neutrino field^. However, the presence of the Majorana 

(see Eq.dOal)). In 



phases does not have any effect on neutrino oscillations I^J, |6^, ^ 
conclusion, the general expression for the mixing matrix can be written as 



U = V23Vi3Wi2D{\). 



(A.19) 



Explicitly we have 



U 



-Sl2C23e' 
Y Si2S23e" 



Cl2Cl3e 
-«(f?i2-Ai) 



" Ci2S23Si3e 
Cl2C23Sl3e 



Cl2C23e 



Sl2Cl3e 
i\2 



■S13 



512323^136'^"^^+^^^ S23C13 



-Cl2S23e*^" - Sl2C23Sl3e*(''^"+^2) C23C13 



(A.20) 



^ For instance, the Majorana mass term for one left-handed neutrino field is proportional to 
T^lvI + Vi^vl = ~^\fivL ~ '^L^^'^L, where vl — CV^ and C is the charge conjugation matrix, ft is clear 
that the Majorana mass term is not invariant under the rephasing v 



e '■^v. 
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with Cij = cos and Sij = sin.'&ij. This parameterization of the mixing matrix is useful 
for the study of neutrino oscillations in matter (see Section H). 
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Figure Captions 



Figure [l|. Density p (A) and electron number density (B) in the interior of the Earth as 
functions of the radial distance r from the center of the Earth. The solid curves 
represent the data given in Ref.[^]. The electron number density is given by 
AT, = NA{p/g){Z/A) with {Z/A) = 0.475 for r < 3480 Km (core) and {Z/A) = 0.495 
for r > 3480 Km (mantle). The dotted curves represent our approximation in terms 
of five shells with outer radii rj=i ,., 5 = 1221, 3480, 5701, 5971, 6371 Km, having con- 
stant densities Pi=i,...,5 = 13.0, 11.3, 5.0, 3.9, 3.0 g/cm^ and electron number density 
(iVe)i=i,...,5 = 6.15, 5.36, 2.47, 1.93, 1.50 A^Acm-l 

Figure Averaged value ( p.69b|) of the survival probabilities of muon neutrinos (solid 
curves) and anti-neutrinos (dashed curves) as functions of the distance L = t in a 
medium with constant electron density Ne = 3 Nacbi^^. Two sets of values of the 
mixing parameters are considered: (A) |?7e3p = |f^^3p = 1/3 (which corresponds to 
maximal mixing of the three neutrinos) and (B) |t/e3p = 0.3, |?7^3p = 0.5 (which is 
close to the best fit of the Kamiokande data; see Eq.( |3.6| )). The dotted lines represent 
the averaged survival probabilities in vacuum ( p.71|) , which are equal for neutrinos and 
anti-neutrinos. 

Figure Fig.^,B: (^P^^^^^) (solid curves) and {|Pp^^p^) (dashed curves) for neutrinos 
propagating in the interior of the Earth as a function of the zenithal angle 6. These 
figures correspond to the same two sets of values of the mixing parameters as in Fig.^]. 
The dotted lines represent the averaged survival probabilities in vacuum ( p.71| ), which 
are equal for neutrinos and anti-neutrinos. Fig.^: slant depth as a function of the 
zenithal angle 6. 

Figure Region in the Be-e-ml plane allowed at 90% CL by the Kamiokande data 
(the shadowed region). The lightly shadowed region is forbidden by the results of the 
Bugey |^0[ and Krasnoyarsk ||6T| reactor z/g disappearance experiments, whose exclusion 



curves are represented by the dahed and dotted lines, respectively. The dash-dotted 
and dash-dot-dotted curves represent the sensitivity of the CHOOZ |3l[ and Palo Verde 



3^ reactor long-baseline experiments. 



Figure Region allowed at 90% CL by the Kamiokande data in the B^.^-m\ plane (the 



shadowed region). The dashed line reproduces the exclusion curve of the CDHS |79| 
^Vn disappearance experiment. The dash-dotted and dash-dot-dotted curves represent 



the sensitivity of the K2K and MINOS |Q accelerator long-baseline experiments 



Figure Region in the A^.g-ml plane allowed at 90% CL by the analysis of the 
Kamiokande data (the shadowed region). The lightly shadowed region is excluded 
by the results of the Bugey and Krasnoyarsk |^ reactor neutrino experiments. 



The long-dashed, dash-dotted and dash-dot-dotted curves represent the sensitivities 



of the K2K H, MINOS Q and ICARUS |3|] long-baseline experiments. 
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Figure Region in the Af^-r^m-s plane allowed at 90% CL by the analysis of the 
Kamiokande data (the shadowed region). The lightly shadowed region is indirectly 
excluded by the results of the reactor neutrino oscillation experiments, leaving the 
darkly shadowed allowed region. The dash-dotted and dash-dot-dotted curves rep- 
resent the sensitivities of the MINOS and ICARUS long-baseline experiments in the 
^'u channel. 
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